Large language models encode clinical knowledge.

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.

Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.

Multi-Frequency 3D Shear Wave Absolute Vibro-Elastography (S-WAVE) System for the Prostate.

This article describes a novel system for quantitative and volumetric measurement of tissue elasticity in the prostate using simultaneous multi-frequency tissue excitation. Elasticity is computed by using a local frequency estimator to measure the three-dimensional local wavelengths of steady-state shear waves within the prostate gland. The shear wave is created using a mechanical voice coil shaker which transmits simultaneous multi-frequency vibrations transperineally. Radio frequency data is streamed directly from a BK Medical 8848 transrectal ultrasound transducer to an external computer where tissue displacement due to the excitation is measured using a speckle tracking algorithm. Bandpass sampling is used that eliminates the need for an ultra-fast frame rate to track the tissue motion and allows for accurate reconstruction at a sampling frequency that is below the Nyquist rate. A roll motor with computer control is used to rotate the transducer and obtain 3D data. Two commercially available phantoms were used to validate both the accuracy of the elasticity measurements as well as the functional feasibility of using the system for in vivo prostate imaging. The phantom measurements were compared with 3D Magnetic Resonance Elastography (MRE), where a high correlation of 96% was achieved. In addition, the system has been used in two separate clinical studies as a method for cancer identification. Qualitative and quantitative results of 11 patients from these clinical studies are presented here. Furthermore, an AUC of 0.87±0.12 was achieved for malignant vs. benign classification using a binary support vector machine classifier trained with data from the latest clinical study with leave one patient out cross-validation.

3-D Transducer Mounted Shear Wave Absolute Vibro-Elastography: Proof of Concept.

Quantitative tissue stiffness characterization using ultrasound (US) has been shown to improve prostate cancer (PCa) detection in multiple studies. Shear wave absolute vibro-elastography (SWAVE) allows quantitative and volumetric assessment of tissue stiffness using external multifrequency excitation. This article presents a proof of concept of a first-of-a-kind 3-D hand-operated endorectal SWAVE system designed to be used during systematic prostate biopsy. The system is developed with a clinical US machine, requiring only an external exciter that can be mounted directly to the transducer. Subsector acquisition of radio frequency (RF) data allows imaging of shear waves with a high effective frame rate (up to 250 Hz). The system was characterized using eight different quality assurance phantoms. Due to the invasive nature of prostate imaging, at this early stage of development, validation of in vivo human tissue was instead carried out by intercostally scanning the livers of n = 7 healthy volunteers. The results are compared with 3-D magnetic resonance elastography (MRE) and an existing 3-D SWAVE system with a matrix array transducer (M-SWAVE). High correlations were found with MRE (99% in phantoms, 94% in liver data) and with M-SWAVE (99% in phantoms, 98% in liver data).

Registration of trans-perineal template mapping biopsy cores to volumetric ultrasound.

PURPOSE: Pathology from trans-perineal template mapping biopsy (TTMB) can be used as labels to train prostate cancer classifiers. In this work, we propose a framework to register TTMB cores to advanced volumetric ultrasound data such as multi-parametric transrectal ultrasound (mpTRUS). METHODS: The framework has mainly two steps. First, needle trajectories are calculated with respect to the needle guiding template-considering deflections in their paths. In standard TTMB, a sparsely sampled ultrasound volume is taken prior to the procedure which contains the template overlaid on top of it. The position of this template is detected automatically, and the cores are mapped following the calculated needle trajectories. Second, the TTMB volume is aligned to the mpTRUS volume by a two-step registration method. Using the same transformations from the registration step, the cores are registered from the TTMB volume to the mpTRUS volume. RESULTS: TTMB and mpTRUS of 10 patients were available for this work. The target registration errors (TRE) of the volumes using landmarks picked by three research assistants (RA) and one radiation oncologist (RO) were on average 1.32 ± 0.7 mm and 1.03 ± 0.6 mm, respectively. Additionally, on average, our framework takes only 97 s to register the cores. CONCLUSION: Our proposed framework allows a quick way to find the spatial location of the cores with respect to volumetric ultrasound. Furthermore, knowing the correct location of the pathology will facilitate focal treatment and will aid in training imaging-based cancer classifiers.

Centre-specific autonomous treatment plans for prostate brachytherapy using cGANs.

PURPOSE: In low-dose-rate prostate brachytherapy (LDR-PB), treatment planning is the process of determining the arrangement of implantable radioactive sources that radiates the prostate while sparing healthy surrounding tissues. Currently, these plans are prepared manually by experts incorporating the centre's planning style and guidelines. In this article, we develop a novel framework that can learn a centre's planning strategy and automatically reproduce rapid clinically acceptable plans. METHODS: The proposed framework is based on conditional generative adversarial networks that learn our centre's planning style using a pool of 931 historical LDR-PB planning data. Two additional losses that help constrain prohibited needle patterns and produce similar-looking plans are also proposed. Once trained, this model generates an initial distribution of needles which is passed to a planner. The planner then initializes the sources based on the predicted needles and uses a simulated annealing algorithm to optimize their locations further. RESULTS: Quantitative analysis was carried out on 170 cases which showed the generated plans having similar dosimetry to that of the manual plans but with significantly lower planning durations. Indeed, on the test cases, the clinical target volumes achieving [Formula: see text] of the prescribed dose for the generated plans was on average [Formula: see text] ([Formula: see text] for manual plans) with an average planning time of [Formula: see text] min ([Formula: see text] min for manual plans). Further qualitative analysis was conducted by an expert planner who accepted [Formula: see text] of the plans with some changes ([Formula: see text] requiring minor changes & [Formula: see text] requiring major changes). CONCLUSION: The proposed framework demonstrated the ability to rapidly generate quality treatment plans that not only fulfil the dosimetric requirements but also takes into account the centre's planning style. Adoption of such a framework would save significant amount of time and resources spent on every patient; boosting the overall operational efficiency of this treatment.

Prostate brachytherapy intraoperative dosimetry using a combination of radiographic seed localization with a C-arm and deformed ultrasound prostate contours.

PURPOSE: The purpose of the study was to assess the feasibility of performing intraoperative dosimetry for permanent prostate brachytherapy by combining transrectal ultrasound (TRUS) and fluoroscopy/cone beam CT [CBCT] images and accounting for the effect of prostate deformation. METHODS AND MATERIALS: 13 patients underwent TRUS and multiview two-dimensional fluoroscopic imaging partway through the implant, as well as repeat fluoroscopic imaging with the TRUS probe inserted and retracted, and finally three-dimensional CBCT imaging at the end of the implant. The locations of all the implanted seeds were obtained from the fluoroscopy/CBCT images and were registered to prostate contours delineated on the TRUS images based on a common subset of seeds identified on both image sets. Prostate contours were also deformed, using a finite-element model, to take into account the effect of the TRUS probe pressure. Prostate dosimetry parameters were obtained for fluoroscopic and CBCT-dosimetry approaches and compared with the standard-of-care Day-0 postimplant CT dosimetry. RESULTS: High linear correlation (R2 > 0.8) was observed in the measured values of prostate D90%, V100%, and V150%, between the two intraoperative dosimetry approaches. The prostate D90% and V100% obtained from intraoperative dosimetry methods were in agreement with the postimplant CT dosimetry. Only the prostate V150% was on average 4.1% (p-value <0.05) higher in the CBCT-dosimetry approach and 6.7% (p-value <0.05) higher in postimplant CT dosimetry compared with the fluoroscopic dosimetry approach. Deformation of the prostate by the ultrasound probe appeared to have a minimal effect on prostate dosimetry. CONCLUSIONS: The results of this study have shown that both of the proposed dosimetric evaluation approaches have potential for real-time intraoperative dosimetry.

Focal application of low-dose-rate brachytherapy for prostate cancer: a pilot study.

PURPOSE: To evaluate the feasibility and to report the early outcomes of focal treatment of prostate cancer using low-dose-rate brachytherapy (LDR-PB). MATERIAL AND METHODS: Seventeen patients were screened with multi-parametric magnetic resonance imaging (mpMRI), 14 of whom proceeded to receive trans-perineal template mapping biopsy (TTMB). Focal LDR-PB was performed on five eligible patients using dual air kerma strength treatment plans based on planning target volumes derived from cancer locations and determined by TTMB. Patient follow-up includes prostate specific antigen (PSA) measurements, urinary and sexual function questionnaires, repeated imaging and TTMB at specific intervals post-treatment. RESULTS: Feasibility of focal LDR-PB was shown and short-term outcomes are promising. While the detection rate of tumors, a majority of which were low grade GS 3 + 3, was found to be low on mpMRI (sensitivity of 37.5%), our results suggest the potential of mpMRI in detecting the presence of higher grade (GS ≥ 3 + 4), and bilateral disease indicating its usefulness as a screening tool for focal LDR-PB. CONCLUSIONS: Low-dose-rate brachytherapy is a favorable ablation option for focal treatment of prostate cancer, requiring minimal modification to the standard (whole gland) LDR-PB treatment, and appears to have a more favorable side effect profile. Further investigation, in the form of a larger study, is needed to assess the methods used and the long-term outcomes of focal LDR-PB.

Robustness to source displacement in dual air kerma strength planning for focal low-dose-rate brachytherapy of prostate cancer.

PURPOSE: To describe the use of dual source strength implants for focal low-dose-rate brachytherapy. METHODS AND MATERIALS: An interneedle dual source strength planning strategy is described for focal low-dose-rate brachytherapy of the prostate. The implanted treatment plans were designed using peripheral (except near the rectum) needles loaded with high strength (0.9 U) sources and central needles loaded with low strength (0.4 U) sources ("interneedle" dual strength planning). This approach has been applied for focally treating 3 patients. In this article, we compare the characteristics and robustness to source motion of interneedle dual strength planning with four alternative planning strategies (single strength high, low, and intermediate, and intraneedle dual strength) on 50 simulated cases. RESULTS: Interneedle dual source strength planning results in greater robustness to source motion and overall lower seed and needle density compared to the standard low source strength planning currently used in our centre. This planning approach is also significantly superior to single strength high, single strength intermediate and intraneedle dual strength planning strategies in terms of high dose to the urethral avoidance structure. CONCLUSIONS: The use of interneedle dual source strength treatment plans for focal low-dose-rate brachytherapy is possibly the practical solution for limiting the density of sources required to deliver the prescribed dose while limiting proximity of high strength sources to organs at risk.

A multi-atlas-based segmentation framework for prostate brachytherapy.

Low-dose-rate brachytherapy is a radiation treatment method for localized prostate cancer. The standard of care for this treatment procedure is to acquire transrectal ultrasound images of the prostate in order to devise a plan to deliver sufficient radiation dose to the cancerous tissue. Brachytherapy planning involves delineation of contours in these images, which closely follow the prostate boundary, i.e., clinical target volume. This process is currently performed either manually or semi-automatically, which requires user interaction for landmark initialization. In this paper, we propose a multi-atlas fusion framework to automatically delineate the clinical target volume in ultrasound images. A dataset of a priori segmented ultrasound images, i.e., atlases, is registered to a target image. We introduce a pairwise atlas agreement factor that combines an image-similarity metric and similarity between a priori segmented contours. This factor is used in an atlas selection algorithm to prune the dataset before combining the atlas contours to produce a consensus segmentation. We evaluate the proposed segmentation approach on a set of 280 transrectal prostate volume studies. The proposed method produces segmentation results that are within the range of observer variability when compared to a semi-automatic segmentation technique that is routinely used in our cancer clinic.

Multiparametric 3D in vivo ultrasound vibroelastography imaging of prostate cancer: Preliminary results.

PURPOSE: Ultrasound-based solutions for diagnosis and prognosis of prostate cancer are highly desirable. The authors have devised a method for detecting prostate cancer using a vibroelastography (VE) system developed in our group and a tissue classification approach based on texture analysis of VE images. METHODS: The VE method applies wide-band mechanical vibrations to the tissue. Here, the authors report on the use of this system for cancer detection and show that the texture of VE images characterized by the first and the second order statistics of the pixel intensities form a promising set of features for tissue typing to detect prostate cancer. The system was used to image patients prior to radical surgery. The removed specimens were sectioned and studied by an experienced histopathologist. The authors registered the whole-mount histology sections to the ultrasound images using an automatic registration algorithm. This enabled the quantitative evaluation of the performance of the authors' imaging method in cancer detection in an unbiased manner. The authors used support vector machine (SVM) classification to measure the cancer detection performance of the VE method. Regions of tissue of size 5 × 5 mm, labeled as cancer and noncancer based on automatic registration to histology slides, were classified using SVM. RESULTS: The authors report an area under ROC of 0.81 ± 0.10 in cancer detection on 1066 tissue regions from 203 images. All cancer tumors in all zones were included in this analysis and were classified versus the noncancer tissue in the peripheral zone. This outcome was obtained in leave-one-patient-out validation. CONCLUSIONS: The developed 3D prostate vibroelastography system and the proposed multiparametric approach based on statistical texture parameters from the VE images result in a promising cancer detection method.

Semiautomatic segmentation for prostate brachytherapy: dosimetric evaluation.

PURPOSE: To demonstrate that manual prostate segmentation in transrectal ultrasound images can be replaced with semiautomatic segmentation. METHODS AND MATERIALS: Semiautomatic segmentation using a tapered ellipsoid model was applied to transrectal ultrasound images. Region-based volumetric evaluation was performed between original and physician-reviewed semiautomatic contours. For dosimetric assessment, treatment plans generated on semiautomatic contours were overlaid on physician-reviewed semiautomatic contours and dose parameters were computed. To establish a threshold for the acceptable amount of dosimetric degradation below which the adoption of semiautomatic planning is unacceptable, the range of variability in dosimetric quality attributed to manual variability was obtained and compared with that of semiautomatic contours. RESULTS: An average volume error (1-Dice similarity coefficient) of less than 7% between semiautomatic and manual volumes (140 cases) was obtained. The difference between the mean V(100) of plans created for semiautomatic contours then overlaid on physician-reviewed semiautomatic contours and the original V(100) values, that is, before overlaying on the physician-reviewed contours (41 cases) was lower than 5%. An average total duration of 2-4min, which includes algorithm initialization, 11.67±3.57s algorithm time, and contour modification is required per case. This algorithm is being used at the British Columbia Cancer Agency and to this date has been applied for the treatment of more than 600 patients. CONCLUSIONS: In terms of volumetric and dosimetric accuracy, the proposed algorithm is a suitable replacement for manual segmentation in the context of our planning technique. The benefits are shorter segmentation times; greater consistency; less reliance on user experience; and smooth, symmetric contours.

An automatic multi-atlas segmentation of the prostate in transrectal ultrasound images using pairwise atlas shape similarity.

Delineation of the prostate from transrectal ultrasound images is a necessary step in several computer-assisted clinical interventions, such as low dose rate brachytherapy. Current approaches to user segmentation require user intervention and therefore it is subject to user errors. It is desirable to have a fully automatic segmentation for improved segmentation consistency and speed. In this paper, we propose a multi-atlas fusion framework to automatically segment prostate transrectal ultrasound images. The framework initially registers a dataset of a priori segmented ultrasound images to a target image. Subsequently, it uses the pairwise similarity of registered prostate shapes, which is independent of the image-similarity metric optimized during the registration process, to prune the dataset prior to the fusion and consensus segmentation step. A leave-one-out cross-validation of the proposed framework on a dataset of 50 transrectal ultrasound volumes obtained from patients undergoing brachytherapy treatment shows that the proposed is clinically robust, accurate and reproducible.

Seed localization in ultrasound and registration to C-arm fluoroscopy using matched needle tracks for prostate brachytherapy.

We propose a novel fiducial-free approach for the registration of C-arm fluoroscopy to 3-D ultrasound images of prostate brachytherapy implants to enable dosimetry. The approach involves the reliable detection of a subset of radioactive seeds from 3-D ultrasound, and the use of needle tracks in both ultrasound and fluoroscopy for registration. Seed detection in ultrasound is achieved through template matching in 3-D radio frequency ultrasound signals, followed by thresholding and spatial filtering. The resulting subset of seeds is registered to the complete reconstruction of the brachytherapy implant from multiple C-arm fluoroscopy views. To compensate for the deformation caused by the ultrasound probe, simulated warping is applied to the seed cloud from fluoroscopy. The magnitude of the applied warping is optimized within the registration process. The registration is performed in two stages. First, the needle track projections from fluoroscopy and ultrasound are matched. Only the seeds in the matched needles are then used as fiducials for point-based registration. We report results from a physical phantom with a realistic implant (average postregistration seed distance of 1.6 ± 1.2 mm) and from five clinical patient datasets (average error: 2.8 ± 1.5 mm over 128 detected seeds). We conclude that it is feasible to use RF ultrasound data, template matching, and spatial filtering to detect a reliable subset of brachytherapy seeds from ultrasound to enable registration to fluoroscopy for dosimetry.

Fusion of ultrasound B-mode and vibro-elastography images for automatic 3D segmentation of the prostate.

Prostate segmentation in B-mode images is a challenging task even when done manually by experts. In this paper we propose a 3D automatic prostate segmentation algorithm which makes use of information from both ultrasound B-mode and vibro-elastography data.We exploit the high contrast to noise ratio of vibro-elastography images of the prostate, in addition to the commonly used B-mode images, to implement a 2D Active Shape Model (ASM)-based segmentation algorithm on the midgland image. The prostate model is deformed by a combination of two measures: the gray level similarity and the continuity of the prostate edge in both image types. The automatically obtained mid-gland contour is then used to initialize a 3D segmentation algorithm which models the prostate as a tapered and warped ellipsoid. Vibro-elastography images are used in addition to ultrasound images to improve boundary detection.We report a Dice similarity coefficient of 0.87±0.07 and 0.87±0.08 comparing the 2D automatic contours with manual contours of two observers on 61 images. For 11 cases, a whole gland volume error of 10.2±2.2% and 13.5±4.1% and whole gland volume difference of -7.2±9.1% and -13.3±12.6% between 3D automatic and manual surfaces of two observers is obtained. This is the first validated work showing the fusion of B-mode and vibro-elastography data for automatic 3D segmentation of the prostate.

Evaluation of visualization of the prostate gland in vibro-elastography images.

In this paper, vibro-elastography (VE), an ultrasound-based method that creates images of tissue viscoelasticity contrast, is evaluated as an imaging modality to visualize and segment the prostate. We report a clinical study to characterize the visibility of the prostate in VE images and the ability to detect the boundary of the gland. Measures for contrast, edge strength characterized by gradient and statistical intensity change at the edge, and the continuity of the edges are proposed and computed for VE and B-mode ultrasound images. Furthermore, using MRI as the gold standard, we compare the error in the computation of the volume of the gland from VE and B-mode images. The results demonstrate that VE images are superior to B-mode images in terms of contrast, with an approximately six fold improvement in contrast-to-noise ratio, and in terms of edge strength, with an approximately two fold improvement in the gradient in the direction normal to the edge. The computed volumes show that the VE images provide an accurate 3D visualization of the prostate with volume errors that are slightly lower than errors computed based on B-mode images. The total gland volume error is 8.8±2.5% for VE vs. MRI and 10.3±4.6% for B-mode vs. MRI, and the total gland volume difference is -4.6±11.1% for VE vs. MRI and -4.1±17.1% for B-mode vs. MRI, averaged over nine patients and three observers. Our results show that viscoelastic mapping of the prostate region using VE images can play an important role in improving the anatomic visualization of the prostate and has the potential of becoming an integral component of interventional procedures such as brachytherapy.

Semi-automatic segmentation for prostate interventions.

In this paper we report and characterize a semi-automatic prostate segmentation method for prostate brachytherapy. Based on anatomical evidence and requirements of the treatment procedure, a warped and tapered ellipsoid was found suitable as the a-priori 3D shape of the prostate. By transforming the acquired endorectal transverse images of the prostate into ellipses, the shape fitting problem was cast into a convex problem which can be solved efficiently. The average whole gland error between non-overlapping volumes created from manual and semi-automatic contours from 21 patients was 6.63 ± 0.9%. For use in brachytherapy treatment planning, the resulting contours were modified, if deemed necessary, by radiation oncologists prior to treatment. The average whole gland volume error between the volumes computed from semi-automatic contours and those computed from modified contours, from 40 patients, was 5.82 ± 4.15%. The amount of bias in the physicians' delineations when given an initial semi-automatic contour was measured by comparing the volume error between 10 prostate volumes computed from manual contours with those of modified contours. This error was found to be 7.25 ± 0.39% for the whole gland. Automatic contouring reduced subjectivity, as evidenced by a decrease in segmentation inter- and intra-observer variability from 4.65% and 5.95% for manual segmentation to 3.04% and 3.48% for semi-automatic segmentation, respectively. We characterized the performance of the method relative to the reference obtained from manual segmentation by using a novel approach that divides the prostate region into nine sectors. We analyzed each sector independently as the requirements for segmentation accuracy depend on which region of the prostate is considered. The measured segmentation time is 14 ± 1s with an additional 32 ± 14s for initialization. By assuming 1-3 min for modification of the contours, if necessary, a total segmentation time of less than 4 min is required, with no additional time required prior to treatment planning. This compares favorably to the 5-15 min manual segmentation time required for experienced individuals. The method is currently used at the British Columbia Cancer Agency (BCCA) Vancouver Cancer Centre as part of the standard treatment routine in low dose rate prostate brachytherapy and is found to be a fast, consistent and accurate tool for the delineation of the prostate gland in ultrasound images.

Automatic prostate segmentation using fused ultrasound B-mode and elastography images.

In this paper we propose a fully automatic 2D prostate segmentation algorithm using fused ultrasound (US) and elastography images. We show that the addition of information from mechanical tissue properties acquired from elastography to acoustic information from B-mode ultrasound, can improve segmentation results. Gray level edge similarity and edge continuity in both US and elastography images deform an Active Shape Model. Comparison of automatic and manual contours on 107 transverse images of the prostate show a mean absolute error of 2.6 +/- 0.9 mm and a running time of 17.9 +/- 12.2 s. These results show that the combination of the high contrast elastography images with the more detailed but low contrast US images can lead to very promising results for developing an automatic 3D segmentation algorithm.